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(57) Abstract: A computer implemented and/or assisted 
system presents one or more end users with a dynamic, 
three-tiered virtual reality environment. The environment 
includes one or more static backgrounds, one or more users 
in the form of avatars, and a real-time or delayed presentation 
of audio and/or visual events through the introduction of 
a data stream. The user, as an avatar, may view the event 
from multiple positions, directions and angles while moving 
about the virtual environment. The content of the data stream 
is not dependent or relative to the actions of the user or 
pre-programmed factors. Rather, the content represents actual, 
real-time or formerly real-time events to provide the users with 
a dynamic, virtual event that may be experienced on the user's 
computer screen. 



WO 01/85293 Al I Bill IflEIIII I IUIJE IIIII fill J D III Hill If HI Hill lllll 1111 llllllf 1111 Ml HI! 



For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



WO 01/85293 PCT/US00/12783 

METHOD AND SYSTEM FOR, PROVIDING A DYNAMIC VIRTUAL 
ENVIRONMENT USING DATA STREAMING 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention generally relates to a 
computer implemented and/or assisted system and 
5 process for displaying dynamic video, audio or 
alphanumeric information in a virtual reality * 
environment using data streaming. 



Background of the Related Art 

10 Virtual reality ( w VR n ) is a computer- implemented 

and/or assisted representation of reality in three 
dimensions. The use of VR often revolves around 
"tele-existence, " or the projection of one or more 
individuals {referred to as "avatars") into a quasi- 

15 realistic predetermined environment, as well as the 
maintenance of consistency within the virtual 
environment. VR environments may also include one or 
more agents, or entities within the environment whose 
actions are predetermined in response to end-user 

20 actions. VR applications typically focus on either 
inter-avatar activity or avatar/ environment (including 
avatar/agent) interaction. Movement and interaction 
of each avatar is caused by user-initiated actions. 
For example, in a video game, players may often 

25 compete against each other as avatars, or against pre- 
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defined agents, in one or more of a limited number of 
fixed virtual environments. 

Virtual environments have numerous applications. 
For example, high-profile Internet sites are highly 
5 competitive. The Internet has accelerated the classic 
business problem of attracting and retaining prospects 
or visitors to individual sites, Internet users 
increasingly seek, . and Internet site owners 
increasingly seek to provide, rich, interesting, 

10 dynamic environments to work and play on-line. VR can 
also be useful for business training and operational 
applications . 

Prior art attempts to create or enhance a virtual 
environment may be found in: (i) U.S. Patent No. 

15 5,958,012, to Battat et al . , incorporated herein by 
reference, which describes a system and apparatus for 
visualizing the components of a computer network 
system as a three-dimensional virtual environment; 
(ii) U.S. Patent 'No. 6,036,601, to Heckel , 

20 incorporated herein by reference, which describes a 
non-interactive method of advertising within virtual 
game screen environments on the Internet; and (iii) 
U.S. Patent No. 5,917,495, to Doi et al . , incorporated 
herein by reference, which describes an apparatus and 

25 method for presenting information in a virtual 
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environment in which a user can feel presence, as 
actual space . 

All of the prior art virtual environments have 
several disadvantages. The prior art environments 
5 have either: (i) provided a static environment (e.g., 
a predefined room having predefined walls, windows, 
doors, etc.); or (ii) attempted to provide a "pseudo- 
dynamic" environment by allowing the user to select 
from or move between several static environments. An 

10 example of a static environment is a racetrack game in 
which the user is provided with a single racetrack per' 
game. An example of a "pseudo-dynamic" environment 
may be found in a multi-level video game in which the 
user, as an avatar, moves through several rooms or 

15 levels based on game performance and user-selected 
actions. Such environments, in the consumer context, 
have limited appeal, as consumers may become 
disinterested in a fixed or "pseudo-dynamic" 
environment. In the business context, such 

20 environments have limited use because they do not 
allow for the inclusion of real-time, delayed or 
stored dynamic information into the virtual 
environment . 

Without provisions for the inclusion of dynamic 
25 information, the prior art virtual environment methods 
and/or systems fail to allow for the alteration of the 
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environment without human intervention. They also 
fail to allow for the dynamic translation of real-time 
and/or real-world information or representations into 
the virtual environment. In addition, the prior art 
5 methods and/ or systems of providing virtual 
environments also fail to maximize or optimize 
delivery of data to the user over the available 
bandwidth. 

I have determined that if streaming video, audio, 

10 alphanumeric or binary data were streamed into a 
three-dimensional virtual environment, user enjoyment 
would be enhanced by the richness of the resulting 
dynamic environment. Internet site owners and virtual 
software developers would benefit from the number of 

15 consumers who would seek such environments . 

Developers and users of business applications could 
also benefit from the availability of VR environments 
that can include streaming real-time, delayed- time or 
stored elements. In addition, on-line communities 

20 could be enhanced as numerous users react to and 
interact around a common event. Further, business 
system controls could provide enhanced coordination of 
operations by providing virtual representations of 
actual , dynamic spaces . 

25 Adding a data stream representative of three- 

dimensional real world events to a three-dimensional 
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VR environment provides several advantages- It 
expands the market for VR applications be allowing 
users to experience the presence of other users, as 
avatars, while sharing or experiencing the same event 
5 at the same time. Audio/video events may include, for 
example, concerts, sports, lectures or other group 
events that may interest consumers. When business 
control information or control systems are the data 
source, applications include in-plant control and 

10 near- limitless application in logistics, including 
warehousing, container shipping and freight. 

Accordingly, I have determined that it is 
desirable to provide a method and/or system that 
allows streaming audio, video, alphanumeric and/or 

15 binary data to be included in a three-dimensional 
virtual environment . 

I have also determined that it is desirable to 
provide a method and/or system for providing a virtual 
environment that allows for the alteration of the 

2 0 environment without human intervention. 

I have also determined that it is desirable to 
provide a method and/ or system that allows for the 
dynamic translation of real-time and/or real-world 
information or representations into a virtual 

2 5 environment . 



- 5 - 



WO 01/85293 



PCT/US00/12783 



I have also determined that it is desirable to 
provide a method and/or system for providing a virtual 
environment that maximizes and/or optimizes the 
delivery of data to the user based on the available 
5 bandwidth. 

I have also determined that it is desirable to 
provide a method and/or system for providing a virtual 
environment that allows users to react to and interact 
around a common event. 
10 I have also determined that it is desirable to 

provide a method and/ or system for providing virtual 
representations of actual, dynamic spaces for business 
applications . 

15 SUMMARY OF THE INVENTION 

It is therefore a feature and advantage of the 
present invention to provide a method and /or system 
that allows streaming audio, video, alphanumeric 
and/or binary data to be included in a three- 

20 dimensional virtual environment. 

It is another feature and advantage of the 
present invention to provide a method and/ or system 
for providing a virtual environment that allows for 
the alteration of the environment without human 

25 intervention. 
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It is another feature and advantage of the 
present invention to provide a method and/ or system 
for providing a virtual environment that maximizes 
and/or optimizes the delivery of data to the user 
5 based on the available bandwidth. 

It is another feature and advantage of the 
present invention to provide a method and/or system 
that allows for the dynamic translation of real-time 
and/or real-world information or representations into 
10 a virtual environment. 

It is another feature and advantage of the 
present invention to provide a method and/or system 
for providing a virtual environment that allows users 
to react to and interact around a common event. 
15 It is another feature and advantage of the 

present invention to provide a method and/or system 
for providing virtual representations of actual, 
dynamic spaces for business applications. 

The above and other features and advantages are 
2 0 achieved though the use of a novel virtual environment 
presentation system and method as herein disclosed. 

In accordance with the preferred embodiment of the 
present invention, streaming data representative of 
video images and/ or audio sound and /or alphanumeric 
25 text is collected from the real world by one or more 
sensing devices. The streaming data is translated 
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into a video, audio or text format for display using 
a computer system equipped with a browser or 
equivalent software. The computer and browser present 
a predetermined, three-dimensional, virtual 

5 environment to the user and also present the 
translated streaming data to the user within the 
virtual environment. Graphical video images, and 
optionally representations of text, are presented in 
a three-dimensional format. 

10 There has thus been outlined, rather broadly, the 

more important features of the invention in order that 
the detailed description thereof that follows may be 
better understood, and in order that the present 
contribution to the art .may be better appreciated. 

15 There are, of course, additional features of the 
invention that will be described hereinafter and which 
will form the subject matter of the claims appended 
hereto . 

In this respect, before explaining at least one 
2 0 embodiment of the invention in detail, it is to be 
understood that the invention is not limited in its 
application to the details of construction and to the 
arrangements of the components set forth in the 
following description or illustrated in the drawings. 
2 5 The invention is capable of other embodiments and of 
being practiced and carried out in various ways. 
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Also, it is to be understood that the phraseology and 
terminology employed herein are for the purpose of 
description and should not be regarded as limiting. 

As such, those skilled in the art will appreciate 
5 that the conception upon which this disclosure is 
based may readily be utilized as a basis for the 
designing of other structures, methods and systems for 
carrying out the several purposes of the present 
invention. It is iirportant, therefore, that the claims 

10 be regarded as including such equivalent constructions 
insofar as they do not depart from the spirit and 
scope of the present invention. 

Further, the purpose of the foregoing abstract is 
to enable the U.S. Patent and Trademark Office and the 

15 public generally, and especially the scientists, 
engineers and practitioners in the art who are not 
familiar with patent or legal terms or phraseology, to 
determine quickly from a cursory inspection the nature 
and essence of the technical disclosure of the 

2 0 application. The abstract is neither intended to 
define the invention of the application, which is 
measured by the claims, nor is it intended to be 
limiting as to the scope of the invention in any way. 
The scope of the invention, together with other 

25 objects of the invention, along with the various 
features of novelty which characterize the invention, 
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are pointed out with particularity in the claims 
annexed to and forming a part of this disclosure. For 
a better under standing of the invention, its operating 
advantages and the specific objects attained by its 
5 uses, reference should be had to the accompanying 
drawings and descriptive matter which illustrate 
preferred embodiments of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
10 FIG. 1 illustrates a computer of a type suitable 

for implementing and/ or assisting in the 
implementation of the processes described herein; 

FIG. 2 is a block diagram illustrating the 
components of a central computer system, which is used 
15 in a preferred embodiment of the present invention; 

FIG. 3 is a block diagram of the primary 
components of the system process; 

FIG. 4 illustrates the steps of translating the 
data stream for presentation to the user or users in 
2 0 a virtual environment; and 

FIG. 5 illustrates a client/server or 
user/network relationship of a type suitable for 
implementing and/or assisting in the implementation of 
the processes described herein. 

25 

Notations and Nomenclature 
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The detailed descriptions which follow may be 
presented in terms of program procedures executed on 
a computer or network of computers. These procedural 
descriptions and representations are the means used by 
5 those skilled in the art to most effectively convey 
the substance of their work to others skilled in the 
art. 

A procedure is here, and generally, conceived to 
be a self-consistent sequence of steps leading to a 

10 desired result. These steps are those requiring 
physical manipulations of physical quantities. 
Usually, though not necessarily, these quantities take 
the form of electrical or magnetic signals capable of 
being stored, transferred, combined, compared and 

15 otherwise manipulated. It proves convenient at times, 
principally for reasons of common usage, to refer to 
these signals as bits, values, elements, symbols, 
characters, terms, numbers, or the like. It should be 
noted, however, that all of these and similar terms 

20 are to be associated with the appropriate physical 
quantities and are merely convenient labels applied to 
these quantities. 

Further, the manipulations performed are often 
referred to in terms, such as adding or comparing, 

25 which are commonly associated with mental operations 
performed by a human operator. While the present 
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invention contemplates the use of an operator to 
access the invention, a human operator is not 
necessary, or desirable in most cases, to perform the 
actual functions described herein which form part of 
5 the present invention; the operations are machine 
operations. Useful machines for performing the 
operation of the present invention include general 
purpose digital computers or similar devices . 

The present invention also relates to an 

10 apparatus for performing these operations. This 
apparatus may be specially constructed for the 
required purpose or it may comprise a general purpose 
comptiter as selectively activated or reconfigured by 
a computer program stored in the computer. The 

15 procedures presented herein are not inherently related 
to a particular computer or other apparatus . Various 
general purpose machines may be used with programs 
written in accordance with the teachings herein, or it 
may prove more convenient to construct a more 

2 0 specialized apparatus to perform the required method 
steps. The required structure for a variety of these 
machines will appear from the description given ♦ 

DESCRIPTION OF THE PREFERRED EMBODIMENT OF THE 
25 INVENTION 
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The present invention is a computer implemented 
and/or assisted system and process for displaying 
dynamic video, audio or alphanumeric information in a 
virtual reality environment using data streaming. The 
5 invention requires a server or network of serving 
computers, one or more user computers, and a means for 
communicating between the server or network and the 
user computer. 

FIG. 1 illustrates a computer of a type suitable 

10 for carrying out and/ or comprising the system of the 
invention. Viewed externally in FIG.l, .a computer 
system designated by reference numeral 101 has a 
central processing unit and disk drives 103 and 104. 
Disk drives 103 and 104 are merely symbolic of a 

15 number of disk drives which might be accommodated by 
the computer system. Typically these would include a 
hard disk drive and optionally one or more floppy disk 
drives such as 103 and/or one or more CD-ROMs, CD-Rs, 
CD-RWs or digital video disk devices indicated by slot 

2 0 104. The number and types of drives typically varies 
with different computer configurations. Disk drives 
103 and 104 are in fact options, and they may be 
omitted from the computer system used in connection 
with the processes described herein. 

25 The computer also has a display 105 upon which 

graphical, video and/or alphanumeric information is 
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displayed. The display may be any device capable of 
presenting visual images, such as a computer monitor, 
a projection device, or even a device such as a 
headset or helmet worn by the user to present visual 
5 images to the user's eyes. The display is optional 
for the server or the network of computers described 
herein. An optional keyboard 106 and a directing 
device 107 such as a mouse, joystick, touch pad, track 
ball, steering wheel, remote control or any other type 

10 of pointing or directing device may be provided as 
input devices to interface with the central processing 
unit. To increase input efficiency, the keyboard 106 
may be supplemented with or replaced by a scanner, 
card reader or other data input device. 

15 Optionally the computer and display may exist in 

the format of a television and a computing device such 
as a set top box that performs the functions of a 
computer. The computing device receives inputs from 
the user's directional device and presents video 

20 and/ or audio formatted data to the user via the 
television screen. 

FIG. 2 illustrates a block diagram of the 
internal hardware of the computer of FIG. 1. A bus 
256 serves as the main information highway 

25 interconnecting the other components of the computer. 
CPU 258 is the central processing unit of the system, 
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performing calculations and logic operations required 
to execute a program. Read only memory (ROM) 260 and 
random access memory (RAM) 262 constitute the main 
memory of the computer. 
5 A disk controller 264 interfaces one or more disk 

drives to the system bus 256. These disk drives may 
be external or internal floppy disk drives such as 
270, external or internal CD-ROM, CD-R, CD-RW or 
digital video disk drives such as 266, or external or 

10 internal hard drives 268. As indicated previously, 
these various disk drives and disk controllers are 
optional devices. 

A display interface 272 permits information from 
the bus 256 to be displayed on the display 248 in 

15 audio or alphanumeric format. Again as indicated, the 
display 248 is also an optional accessory for the 
network or server. The user's display may optionally 
be a television device, ■ computer monitor, LCD display, 
headset, or other device. An audio interface 280 such 

2 0 as a sound card permits information from the bus 25 6 
to be presented in audio format using one or more 
speakers 282 or other audio presentation devices such 
as headphones and the like. Communication with' 
external devices may optionally occur using 

2 5 communication port 274. 
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In addition to the standard components of the 
computer, the computer also includes an interface 254 
which allows for data input through the keyboard 250 
or other input device and/ or the directional or 
5 pointing device 252 such as a mouse or joystick. 

FIG. 3 illustrates the primary components of the 
method for providing a dynamic virtual environment. 
Real world inf ormation 301 having a visual component 
and ah optional audio component is collected using one 

10 or more input or collecting or sensing devices 302. 
Ihe real world information 301 is data representative 
of a dynamic, event, which may include, for example, 
the audio and visual elements of a concert, a lecture, 
a sporting event, or a remote room, area or other 

15 location such as a road, port, warehouse, building, 
city, or an underground tunnel or mine. Optionally, 
the real world information may also be virtual or 
animated presentations that are representative of or 
a simulation of real world information. The one or 

20 more input or collecting or sensing devices 302 may 
include, for example, a camera, microphone, infrared 
sensing device, scanner, bar coding device, sonic 
sensing device, heat sensitive device, and/or any 
other means capable of gathering data representative 

25 of real world objects, actions and events. 
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The input and/ or sensing device transforms the 
real world information into a data stream suitable for 
computer transmission, storage and/or use, such as 
streamable alphanumeric or binary or digital data. 
5 The data representative of real world information is 
optionally stored in a storage device 303, which may 
be located on the server, on one or more of the 
network computers, or on the user's computer. The 
user's computer includes a means for transforming 

10 streaming data into presentable audio, visual and/or 
alphanumeric format such as a browser 304 and one or 
more optional plug-ins having such functionality. 

A three-dimensional virtual environment 306 is 
optionally and preferably stored in storage 303 on, 

15 and/ or generated by, a server or network computer that 
performs housekeeping tasks necessary to maintain the 
predefined virtual environment consistently to one or 
more end users. The virtual environment may optionally 
include one or more agents (i.e./ entities within the 

20 virtual environment that are either predetermined or 
generated by the virtual environment) whose actions 
are predetermined in response to the actions of the 
user or users. The housekeeping tasks may include, 
for example, determining the information each end user 

25 needs to see a display that accurately reflects the 
current state and portion of the environment and the 
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user's position within the environment; placing the 
information into a prioritized queue; monitoring 
communication of the information; receiving 
information from user's browsers 304; and/or 
5 processing or forwarding for processing avatar and 
agent . inf ormation . 

The three-dimensional virtual environment 306 is 
presented to the user on a display device via the 
browser 3 04 or equivalent software having the 

10 capability of presenting visual information to the 
user. The browser receives and interprets information 
from the server and/or network computer, resulting in 
a depiction of the three-dimensional virtual 
environment on the user's computer system display. 

15 The browser is optionally and preferably equipped 
and/or supplemented with one or more plug-in 
applications that assist in the presentation of real- 
time, streaming data in audio and/or visual format. 

One or more users, represented by avatars in the 

20 virtual environment, experience the streaming data 
within the virtual environment 307 and move about the 
environment via a graphical user interface (GUI) 305 
using the computer's directional and/or input device 
such as a joystick, keyboard, mouse, touch pad, or 

25 other device. The GUI 405 may respond to the 
user's /avatar's movements by directing the input or 
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sensing devices 302 to retrieve the streaming data 
representative of real world information from a 
different angle, position or direction 308. The user 
may then experience the data stream in three 
5 dimensions as the user moves about the virtual 
environment. The data stream is periodically or 
continually updated as the input or sensing device 
collects additional information. 

Optionally, the input/sensing device 3 02 

10 automatically collects the streaming data 
representative of real world information 301 from 
multiple positions, angles or directions, and the GUI 
and browser present to the user only the portion of 
such data that is appropriate for the user to 

15 experience based on the user's position within the 
virtual environment . 

FIG. 4 illustrates the steps of translating the 
data stream for presentation to the user or users in 
the virtual environment. The data stream 401 is 

20 gathered by the input and/ or sensing device and pre- 
processed 402 or converted into a computer-readable 
and/ or transmissible format such as binary, 
alphanumeric or other digital code. Optionally, 
certain requirements 403 such as user-responsive 

25 directional or angle of view requirements may be 
included in the pre-processing step. The data stream 
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may also be modified, restricted or otherwise 
processed to include specific properties 404, such as 
size, shape, color, and/or duration limitations to 
accommodate placement of the data stream into the 
5 virtual environment. The data stream is then 
translated into a format that may be likened to that 
of a self -modifying agent and placed into the virtual 
environment 405. The virtual environment is then 
presented 406 to the user or users to include the 

10 predetermined virtual environment and the data stream 
transformed as a self -modifying agent in a manner that 
is responsive to the actions of the user or users. 

FIG. 5 is a block diagram of an additional 
embodiment of the hardware design of the present 

15 invention using a client/ server or user/network 
relationship. An input or sensing device 502 collects 
real world information 5 01 and transforms the 
information into a format suitable for computer 
transmission, storage and/or use, such as into binary, 

20 digital or alphanumeric data. The sensing device 502 
transmits the transformed information to a mainframe, 
server or central computer 5 02 through a 
communications link 503, which may, for example, be a 
telephone and telephone line, the Internet, a wireless 

25 or satellite transmission, or a cable. Optionally, the 
sensing or input device 502 may be connected directly 
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to or integral with the central computer 504, thus 
eliminating the need for a communications link or 
replacing the communications link with a direct 
connection. Optionally, the transformed information 
5 may be stored in a storage means, which may be part of 
or connected to the sensing or input device 502, the 
communications link 503, or the central computer 5 04. 

A three-dimensional virtual environment 510 is 
presented to one or more users on each user's computer 

10 507-509 through one of two methods. First, the 
virtual environment 510 may be generated at or by the 
user's computer 507-509 based on information contained 
in a storage medium such as a hard drive, floppy disk, 
CD-ROM or the like as represented in FIGs . 1 and 2 . 

15 In the alternative, the virtual environment 510 may 
be generated at or by the mainframe, server or central 
computer 5 04 and transmitted to each user's computer 
507-509 through a communications link 506. The 
communications link 506 may be, for example, a 

20 telephone and telephone line, the Internet, a direct 
cable connection, a wireless or cellular or satellite 
transmission, a cable and cable modem, or any device 
capable of transmitting the virtual environment from 
the server to each user. Optionally, a security layer 

25 505 is used to ensure that only users who are 
authorized to receive the virtual environment actually 
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receive the virtual environment. The security layer 
may include, for example, a firewall, a password entry 
and verification process, an automated identity 
.verification process such as the use of "cookies," or 
5 any means of ensuring that only authorized users 
receive the virtual environment. 

The transformed real world information is also 
transmitted from the server 505 to each user 507-509 
via the communication link 5 06. The coinmuni cations 

10 link 506 may be, for example, a telephone and 
telephone line and modem, the Internet, a direct cable 
connection, a cable and cable modem, a wireless or 
cellular or satellite transmission, or any device 
capable of transmitting the transformed real world 

15 information from the server to each user. Optionally, 
a security layer 505 is used to ensure that only users 
who are authorized to receive the transformed real 
world information actually receive the transformed 
real world information. The security layer may 

20 include, for example, a firewall, a password entry and 
verification process, an automated identity 
verification process such as the use of "cookies, " or 
any means of ensuring that only authorized users 
receive the trans formed real world information. Each 

25 user may select an individual view of the virtual 
environment 510 and/or real world information by 
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sending a command, via an input or directional device 
at the user's computer, to the server through the 
communications link. The real world information is 
periodically' or continually updated and presented to 
5 each user based on both changes in the information and 
• changes in the view selected by the user. 

As additional explanation, in the present 
invention video images and/ or audio sound and/or 
alphanumeric text are collected from the real world by 

10 one or more sensing devices. The streaming data is 
translated into a video, audio or text format for 
display using a computer system equipped with a 
browser or ecjuivalent software. The computer and 
browser present a predetermined, three-dimensional, 

15 virtual environment to the user and also present the 
translated streaming data to the user within the 
virtual environment. Graphical video images, and 
optionally representations of text, are presented in 
a three-dimensional format. The actions presented by 

20 the data stream are not responsive to the actions of 
the user, but rather are determined by external 
factors, i.e., what is happening or what happened in 
the real world when the input device 3 02 gathered the 
information. The actions presented may be likened to 

25 a selfr^orphing agent, with actions that reflect the 
actual properties of the data stream at any point in 
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time. Unlike prior art agents, however, the self- 
morphing agent's actions are not pre-determined or 
pre-programmed through human intervention (except to 
the optional extent that such actions may include 
5 causing the agent's actions to be stored in storage). 

Examples of Internet or consumer software-based 
environments that can be created or enhanced with data 
streaming include on-line concert forums, on-line 
shopping destinations, and vertical applications such 

10 as creating a virtual alumni viewing area for a 
college sports event or hosting an on-line conference 
or convention. For example, in a virtual concert 
forum, the typical static environment could include 
the three-dimensional forum itself (walls, floor, 

15 stage, etc.)/ and one or more users could enter the 
hall as viewer and/or participant avatars. The users' 
experience would be enhanced if a performer or 
performers (such as a music group) were inserted onto 
the stage through streaming data in a live-action 

20 format. The streaming data may originate, for 
example, from an actual, real-time concert in which 
the performers were being filmed and recorded, the 
audio data and visual data were converted to a digital 
data stream, the digital data stream was inserted into 

25 the virtual environment, and the data stream was 
transformed back to a real-time virtual representation 
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of the band in the virtual concert hall. Instead of 
being a real-time presentation, the data stream could 
be pre-recorded and/or stored and streamed into the 
virtual environment at a later time, either at a 
5 predetermined time, at a time selected by the host of 
the virtual environment, or at a time selected by the 
user. In such a virtual environment, participants can 
optionally interact and share with each other their 
thoughts and/ or reactions to the virtual performance 

10 as it happens. 

When compared with standardized audio/ video 
streams, business control applications have highly 
individualized data streams. End-user environments 
also differ for each business application, resulting 

15 in increased applications engineering requirements. 
Examples of business environments that can be created 
or enhanced with data streaming include operational 
applications and training modules. For example, a 
three-dimensional virtual representation of a port or 

20 river could also include dynamic representations of 
water traffic (e.g., boats, barges, etc.), or other 
items (e.g., cargo being unloaded from a barge) 
through the insertion of streaming data that is 
representative of the traffic and/or items. The data 

25 stream may be overlaid upon existing control systems 
within a VR-based supervisory and/or control system. 
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This reduces project risk and installation complexity 
while providing a uniform, easy-to-use VR interface 
that can be maintained when underlying systems are 
changed. As illustration, for example a container 
5 port operator could view a three-dimensional 
simulation of the entire facility on a computer 
display- The operator could tt see, ff in three 

dimensions within the simulation, the ships that are 
entering the harbor, their relative speeds and their 

10 safety clearances. The movements of the ships would 
be real-world information that enters the virtual 
environment as a data stream. The operator could also 
"see" a three dimensional representation of a ship and 
its cargo, i.e., containers, and could remove the 

15 cargo from the ship at a remote location by selecting 
a three-dimensional representation of the container on 
the deck of the ship and virtually carrying like, via 
the computer, to another location. The operator's 
actions within the virtual environment would be 

20 translated to the real world as commands to move a 
crane and pick up the containers. As the crane and 
the containers move, the data stream and its position 
within the simulation would change to represent the 
present actual location of the crane and the 

25 containers in the real world. 
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The present invention expands the application of 
virtual environments, including virtual reality 
modeling language (VRML) based environments, the Sony 
community place environment and other environments by 
5 moving beyond the concept of user-controlled avatars 
within a static or predetermined set of environments 
to a three-layer model that also contains the results 
of exogenous data streams (whether video, audio, 
alphanumeric, binary , or otherwise) that have been 

10 translated and embedded within the VR *world" without 
active human intervention. 

The present invention may be used with any 
operating system and/or virtual reality standard, 
including but not limited to, VRML, 3D Studio, 

15 ActiveX, Blaxxun3D, BlaChemical Markup Language (CML) , 
cable modems, Community Place, Cosmo, Direct3D, 
Directory Service Mark up Language (DSML) , Document 
Object Model (DOM) , Digital Signal Processor (DSP) , 
Extendable text languages (DirXML/XML/XSL) , Flash, 

20 Internet Explorer, Java, Linux, Maya, Metastream 
(MTS) , , Macintosh OS, Moving Worlds, Netscape, OpenGL, 
Open Worlds, Personal Data Assistint (PDA) , Scalable 
Vector Graphics (SVG) , set top boxes, Sumatra, Unix, 
WEB3D, Windows, World View, X3D, and Worlds. 

25 The invention allows users (in the form of 

avatars) to interact in an environment that accurately 
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reflects the situation described by the exogenous 
information in real or delayed time. The 
transformation that places the data stream into the 
environment can include enhancements to increase user 
5 comprehension such as (but not limited to) : (i) the 
translation of data into a physical representation of 
an object, such as a moving ship, vehicle, person, 
animal or aircraft; and (ii) the presentation of audio 
and/ or visual cues that can represent real work events 

10 and warn of current or impending problems, such as 
schedule problems, danger of collisions of moving 
objects, and the like. The resulting three- tier 
virtual environment (which includes the static 
background, the data stream, and one or more avatars 

15 and/ or agents) provides a constantly updated virtual 
world without any human intervention (other than the 
optional movement by the user to different locations 
within the virtual world, thus triggering the 
presentation of different views of the three- 

20 dimensional background and data stream) - 

The system of using a network or central server 
provides several advantages- For example, it allows 
multiple users to M see" the same virtual environment, 
and thus the virtual events occurring within the 

25 virtual environment, at the same time. By 
transferring the data stream via a server or network, 
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all users receive the same information at the same 
time in a comparable form. Server-side software 
performs the operations necessary to transform the 
data stream into visual or audio presentation format 
5 and then forwards the images and/ or sound to the user 
or users - 

The client/ server model also provides for 
efficient use of bandwidth by reducing the total 
amount information that must be transmitted to end 

10 users at any single time by providing a single focal 
point for all users to send and retrieve information. 

The concept of VR client/ server distributed 
processing also allows for dynamic adjustment to 
account for bandwidth differences. In other words, for 

15 users having high-speed data connections, more 
information may be coded and sent to the user. For 
users having lower speed connections, selected 
information may be further compressed or omitted, 
providing the lower speed user images and /or sound 

20 that is of somewhat lower resolution or quality than 
that received by the high-speed user, but of 
comparable accuracy and currency. 

The many features and advantages of the invention 
are apparent from the detailed specification, and 

25 thus, it is intended by the appended claims to cover 
all such features and advantages of the invention 
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which fall within the true spirits and scope of the 
invention. Further, since numerous modifications and 
variations will readily occur to those skilled in the 
art, it is not desired to limit the invention to the 
5 exact construction and operation illustrated and 
described, and accordingly, all suitable modifications 
and equivalents may be resorted to, falling within the 
scope of the invention. 
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What is claimed is: 

1. An. interactive computer assisted x&ethod of 
providing a three-dimensional virtual environment to » 
a user comprising* 

{a) collecting, using at least one collecting 
device, data representative of a dynamic event; 

(b) generating, using a computer, a throe- 
dimensional virtual environment; 

(c) inserting, using the computer , the data 
representative of a dynamic event into the three- 
dimensional virtual environment; and 

(d) presenting, using a display . device, a 
presentation of the three-dimensional virtual 
environment and the data representative of a dynamic 
event to the user. 

2. The interactive computer assisted method of 
providing a three-dimensional virtual environment 
aa6ording to claim 1, further comprising: 

(a) receiving, using the computer f a command 
from the user to present a different view of the 
three-dimensional virtual environment; 

(b) collecting, using the at least one 
collecting device , additional data representative of 
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a dynamic event from a direction compatible with, the 
different view selected by the user; 

(a) adjusting, using' the coxgputer, the 
presentation of the three-dimensional virtual 
environment to the user responsive to the different 
view selected by the user; 

(d) inserting , using the computer, the 
additional data representative of a dynamic event into 
the three-dimensional virtual environment; and 

(e) presenting, using the display device, a 
presentation of the three-dimensional virtual 
environment and the additional data representative of 
a dynamic event to the user. 

3. 33foe interactive conputer assisted method of 
providing a three-dimensional virtual environment 
according to claim 1, further comprising: 

(a) collecting, using the at least one 
collecting device, additional data representative of 
a dynamic event; 

(b) replacing, using the computer, the data 
representative of a dynamic event with the additional 
data representative of a dynamic event; and 

(c) presenting, using a display device, a 
presentation of the three-dimensional virtual 
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environment and the additional data representative of 
a dynamic event to the user. 

4. 'The interactive couaputer assisted method of 

* 

providing a three-dimensional virtual environment 
according to claim 1, further conflprising storing the 
data representative of a dynamic event in a storage 
device before* inserting the data into the three- 
dimensional virtual environment. 

5. 'The interactive computer assisted method of claim 
1 wherein, after the generating step, a determining 
step determines a portion of the environment that is 
responsive to a user input. 

6. The interactive cOKpputer assisted method of claim 
1 including the additional step of verifying the 
identity of the user. 

7. An interactive cox«puter assisted method of 
providing a three-dimensional virtual environment to 
at least one user, conqprising: 

(a) collecting, using at least one collecting 
device, data representative of a dynamic event; 
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(b) delivering, using a . first communication 
means, the data representative of a dynamic event to 
a central computer; and 

<c) delivering, using a second communication 
mean©/ the data representative of a dynamic event to 
at least one riser computer for insertion into a three- 
dimensional virtual environment; and 

(d) presenting, using a display device , the data 
representative of a dynamic event to at least one 
user. 

8. Ehe interactive computer assisted method of 
providing a t^ee -dimensional virtual environment 
according to claim 7, further co*nprising* 

(a) receiving, by the central computer, a 
command from a requesting user to present a different 
view of the three-dimensional virtual environment; 

(b) collecting, using the at least one 
collecting device, additional data representative of 
a dynamic event from a direction compatible with the 
different view selected by the requesting user; 

( d) inserting , using the computer , the 
additional data representative of a dynamic event into 
the three-dimensional virtual environment ; and 
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(e) presenting, using the display device, a 
presentation of the additional data representative of 
a dynamic evexit to the requesting user. 

9- The interactive conqpufcer assisted method* of 
providing a thr e e -dixnsns ianal virtual environment 
according to claim 7, further comprising: 

(a) collecting, using the at least one 
collecting 

device, additional data representative of a dynamic 
event; 

(h) replacing, using the central ccaraputer f the 
data representative of a dynamic event with the 
additional data representative of a dynamic event; and 

(c) presenting, using a display device, the 
additional data representative of a dynamic event to 
the at least one user. 

10. The interactive oonnputer assisted method of 
providing 

three-dimensional virtual environment according to 
claim 7, £t*rther comprising storing the data 
representative of a dynamic event in a storage device 
before inserting the data into the three-dimensioiial 
virtual environment. 
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11. tftie interactive ooxnputer assisted method of claim 
7 wherein, after the generating step, a determining 
step determines a portion of the environment that is 
responsive to a user input. 

12 . tfhe interactive computer assisted method of claim. 
7 including the additional step of verifying the 
identity of the user* 

13 • A conqputer program memory, storing computer 
instructions to implement a dynamic three-dimensional 
virtual environment pro gram , the computer instructions 
including: 

♦ 

(a) collecting, using at least one collecting 
device , data representative of a dynamic event; 

(b) generating, using a computer, a three- 
dimensional virtual environment; 

(c) inserting, using the computer, the data 
representative of a dynamic event into the three- 
dimensional virtual environment; and 

(d) presenting, using a display device, a 
presentation of the three-dimensional virtual 
environment and the data representative of a dynamic 
event to the user. 



- 36 - 



WO 01/85293 



PCT/US00/12783 



14. TChe computer program meroory according to claim 
13, the computer instructions further inducing: 

(a) receiving* using the coipputer, a command 
from the user to present a different view of the 

5 three-dixnensiojial virtual environment; 

(b) collecting/ using the at least one 
collecting device, additional data representative of 
a dynamic event from a direction compatible with the 
different view selected by. the user; 

10 (c) adjusting/ using the computer/ the 

presentation of the three-dimensional virtual 
environment to the user responsive to the different 
view selected by the user; 

(d) inserting, using the computer, the 
15 additional data representative of a dynamic event into 

the three-dimensional virtual environment? and 

(e) presenting, using the display device, a 
presentation of the thee-dimensional virtual 
environment and the additional data representative of 

20 a dynamic event to the user. 

15. The computer program memory according to claim 
13/ the computer instructions further including: 

<a) collecting, using the at least one 
25 collecting device, additional data representative of 
a dynamic event; 
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(b) replacing , using the corsputer, the data 
representative of a dynamic event with, the additional 
data representative of a dynamic event; and 

(c) presenting, using a display device, a 
5 presentation of the three-diinensioiial virtual 

environment and the additional data representative of 
a dynamic event to the user. 

16. Ehe cojqputer program memory according to claim 
10 13, the computer instructions further including 
storing the data reopresentative of a dynamic event in 
a storage device before inserting the data into the 
three-dimensional virtual environment. 

15 17. The interactive computer assisted method of claiRi 
13 wherein, after the generating step, a determining 
step determines a portion of the environment that is 
responsive to a user input. 

20 18. The interactive canjputer assisted method of claim 
13 including the additional step of verifying the 
identity of the user. 
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